[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.12.0 #34939

Yikun · 2021-12-18T04:17:35Z

What changes were proposed in this pull request?

This patch aims to upgrade kubernetes-client from 5.10.2 to 5.12.0

Hightlight changes:

v5.12.0: Update Fabric8 Kubernetes Model to v1.23.0, bump jackson-datatype-jsr310 to 2.13.1.
v5.11.0: Introduce the Volcano extension, it would be useful for users who want to use Volcano as customized scheduler in Spark on K8S.
v5.11.0: Breaking change: an abstraction layer added over okHttp

There are also some changes to make sure it is compatiable to create http client with custom dispatcher according to fabric8io/kubernetes-client#3663 (comment) suggestion.

Why are the changes needed?

This will bring several bug fixes and improvements (such as volcano support, Fabric8 Kuberentes model 1.23 support), see more in:

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.
Running a spark-pi job to validate manually.

dongjoon-hyun

Is it for Volcano?

Fix #3579: Add support for Volcano extension

Yikun · 2021-12-18T04:23:04Z

There are some breaking changes in the API of k8s-cli v5.11, looks like it breaks to create httpclient with custom dispatcher like what we did here .

So we need to do some changes to make it work.

issue: fabric8io/kubernetes-client#3663

SparkQA · 2021-12-18T04:41:17Z

Test build #146350 has finished for PR 34939 at commit 5b32fdf.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-18T04:52:56Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50824/

Yikun · 2021-12-18T07:19:31Z

Is it for Volcano?

@dongjoon-hyun Yes. The volcano extesnion was introduced in k8s-cli v5.11.

SparkQA · 2021-12-18T10:26:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50828/

SparkQA · 2021-12-18T10:32:13Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50828/

SparkQA · 2021-12-18T12:14:47Z

Test build #146354 has finished for PR 34939 at commit 4811c24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-18T14:20:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50835/

SparkQA · 2021-12-18T14:38:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50835/

SparkQA · 2021-12-18T15:27:13Z

Test build #146360 has finished for PR 34939 at commit 6d7451d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-19T03:20:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50846/

SparkQA · 2021-12-19T04:13:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50846/

Yikun · 2021-12-19T05:06:55Z

Retest this please

SparkQA · 2021-12-19T05:17:08Z

Test build #146372 has finished for PR 34939 at commit 5b1ab40.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-19T06:21:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50847/

SparkQA · 2021-12-19T07:33:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50847/

SparkQA · 2021-12-19T08:11:41Z

Test build #146373 has finished for PR 34939 at commit 5b1ab40.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Yikun · 2021-12-20T03:11:50Z

...ubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala

@@ -68,6 +69,8 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
      .getOption(s"$kubernetesAuthConfPrefix.$CLIENT_KEY_FILE_CONF_SUFFIX")
    val clientCertFile = sparkConf
      .getOption(s"$kubernetesAuthConfPrefix.$CLIENT_CERT_FILE_CONF_SUFFIX")
+    // TODO(SPARK-37687): clean up direct usage of OkHttpClient, see also:


Here are some backgroud on this note (also add the jira: SPARK-37687):

There are some problems (such as IPV6 based cluster support) on okhttpclient v3, but it's a little bit diffcult to upgrade to v4.

Kubernetes client are also consider to support other clients rather than single okhttpclient.

Kubernetes client add a abstract layer to support variety httpclient, okhttp client as one of supported http clients.

So, we'd better to consider to cleanup okhttpclient direct usage in some right time and use the httpclient which kubernetes client diret supported to reduce the potential risk in future upgrade.

SparkQA · 2021-12-20T04:21:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50861/

SparkQA · 2021-12-20T05:20:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50861/

SparkQA · 2021-12-20T06:22:20Z

Test build #146386 has finished for PR 34939 at commit ecd15cb.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Yikun · 2021-12-20T06:59:36Z

cc @dongjoon-hyun @holdenk @attilapiros

Yikun · 2021-12-20T07:02:10Z

retest this please

SparkQA · 2021-12-20T08:31:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50864/

SparkQA · 2021-12-20T09:48:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50864/

SparkQA · 2021-12-20T10:35:24Z

Test build #146389 has finished for PR 34939 at commit ecd15cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Actually, I'm a little reluctant due to the breaking change. Shall we hold on and see the next release?

Yikun · 2021-12-23T00:12:52Z

@dongjoon-hyun Thanks for review. I can understand your concerns for sure, it makes sense to wait for next release to avoid pontential flaky factor of breaking changes.

But there is one thing I need to clarify, even in the next version we will still face these changes to make okhttpclient work. BTW, these changes are changed according to fabric8io/kubernetes-client#3663 (comment) .

klaus-xiong · 2021-12-23T08:29:38Z

@Yikun Do you have a time line for the 'Cleanup direct usage of OkHttpClient' ?

Yikun · 2021-12-23T10:53:13Z

@klaus-xiong I think the deadline depends on when new better httpclient to replace okhttpclient are supported in k8s client, and also this replacement meet the requirement of Spark on K8S.

It bring some potential issue when k8s-cli to stop okhttpclient support (as fabric8io/kubernetes-client#3547 mentioned, OkHttpClient has some known issue and it's not easy to upgrade OkHttpClient), it may not be a good choice if we still keep the direct OkHttpClient usage in Spark.

### What changes were proposed in this pull request? This PR aims to upgrade K8s client library to 5.10.2 from 5.10.1. ### Why are the changes needed? This is a maintenance version upgrade which brings the following fix. - fabric8io/kubernetes-client@4451030 Please note that this is different from #34939 (5.11.1) which has a breaking change. In addition, the above bug fix is in `5.10.2` and `5.11.2`, not in `5.11.1`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CI. Closes #35182 from dongjoon-hyun/SPARK-37884. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to upgrade K8s client library to 5.10.2 from 5.10.1. ### Why are the changes needed? This is a maintenance version upgrade which brings the following fix. - fabric8io/kubernetes-client@4451030 Please note that this is different from apache#34939 (5.11.1) which has a breaking change. In addition, the above bug fix is in `5.10.2` and `5.11.2`, not in `5.11.1`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CI. Closes apache#35182 from dongjoon-hyun/SPARK-37884. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun

Thank you for your patience. Could you update this PR with 5.12.0? After that, I believe it's time to merge.

https://github.com/fabric8io/kubernetes-client/releases/tag/v5.12.0

Update Fabric8 Kubernetes Model to v1.23.0

Yikun · 2022-01-25T01:44:43Z

@dongjoon-hyun Sure, will upgrade soon. Thanks!

dongjoon-hyun · 2022-01-25T02:16:24Z

Thanks!

dongjoon-hyun

+1, LGTM. Thank you, @Yikun .
Merged to master for Apache Spark 3.3.

github-actions bot added the BUILD label Dec 18, 2021

dongjoon-hyun reviewed Dec 18, 2021

View reviewed changes

github-actions bot added the KUBERNETES label Dec 18, 2021

Yikun force-pushed the SPARK-37258 branch from 93c5eec to 4811c24 Compare December 18, 2021 09:31

Yikun changed the title ~~[SPARK-37258][K8S][Build] Upgrade kubernetes-client to 5.11.0.~~ [SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.0 Dec 18, 2021

Yikun force-pushed the SPARK-37258 branch from 6d7451d to 5b1ab40 Compare December 19, 2021 02:30

Yikun marked this pull request as ready for review December 20, 2021 02:45

Yikun marked this pull request as draft December 20, 2021 02:50

Yikun marked this pull request as ready for review December 20, 2021 03:07

Yikun commented Dec 20, 2021

View reviewed changes

Yikun marked this pull request as draft December 20, 2021 06:12

Yikun marked this pull request as ready for review December 20, 2021 06:58

dongjoon-hyun reviewed Dec 22, 2021

View reviewed changes

Yikun changed the title ~~[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.0~~ [SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.1 Dec 27, 2021

Yikun force-pushed the SPARK-37258 branch 2 times, most recently from 1bf6aac to 65118f6 Compare January 1, 2022 00:28

dongjoon-hyun mentioned this pull request Jan 12, 2022

[SPARK-37884][K8S][BUILD] Upgrade kubernetes-client to 5.10.2 #35182

Closed

Yikun force-pushed the SPARK-37258 branch 2 times, most recently from 88a83c3 to 444ccae Compare January 19, 2022 02:30

Yikun changed the title ~~[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.1~~ [SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.2 Jan 19, 2022

dongjoon-hyun requested changes Jan 25, 2022

View reviewed changes

Upgrade kubernetes-client to 5.12.0

1d4333c

Yikun force-pushed the SPARK-37258 branch from 444ccae to 1d4333c Compare January 25, 2022 03:43

Yikun changed the title ~~[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.11.2~~ [SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.12.0 Jan 25, 2022

dongjoon-hyun approved these changes Jan 25, 2022

View reviewed changes

dongjoon-hyun closed this in 18f9e7e Jan 25, 2022

[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.12.0 #34939

[SPARK-37258][K8S][BUILD] Upgrade kubernetes-client to 5.12.0 #34939

Conversation

Yikun commented Dec 18, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Yikun commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

Yikun commented Dec 18, 2021 • edited Loading

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 18, 2021

SparkQA commented Dec 19, 2021

SparkQA commented Dec 19, 2021

Yikun commented Dec 19, 2021

SparkQA commented Dec 19, 2021

SparkQA commented Dec 19, 2021

SparkQA commented Dec 19, 2021

SparkQA commented Dec 19, 2021

Yikun Dec 20, 2021 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

Yikun commented Dec 20, 2021

Yikun commented Dec 20, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

SparkQA commented Dec 20, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Yikun commented Dec 23, 2021 • edited Loading

klaus-xiong commented Dec 23, 2021

Yikun commented Dec 23, 2021 • edited Loading

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Yikun commented Jan 25, 2022

dongjoon-hyun commented Jan 25, 2022

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Yikun commented Dec 18, 2021 •

edited

Loading

Yikun commented Dec 18, 2021 •

edited

Loading

Yikun Dec 20, 2021 •

edited

Loading

Yikun commented Dec 23, 2021 •

edited

Loading

Yikun commented Dec 23, 2021 •

edited

Loading